
TITLE OF THE INVENTION 
IMAGE PROCESSING APPARATUS AND METHOD, AND COMPUTER- 
READABLE MEMORY 

FIELD OF THE INVENTION 
This invention relates to an image processing 
apparatus and method for processing a moving picture, 
and to a computer-readable memory storing the program 
code for this processing. 

BACKGROUND OF THE INVENTION 
When only a specific scene (a part of a TV program 
that begins from a specific title screen of the program, 
or a scene in which a specific person appears) is to be 
extracted from a plurality of moving-picture materials 
gathered by a video camera or VTR, etc., the user 
searches for the desired scene by repeatedly fast- 
forwarding or playing back each moving-picture material, 
writes down the time code of each scene and creates a 
list of time codes. Then, on the basis of the created 
list of time codes, the user performs an editing 
operation such as gathering specific scenes together and 
combining them into one moving picture. 

However, in order to extract a specific scene from 
a plurality of moving-picture materials, the operator 
must perform the manual operations of fast-forward and 



fy 



playback, search for the desired scene and create the 
list of time codes. This requires a great deal of time 
and labor. 

5 SUMMARY OF THE INVENTION 

Accordingly, an object of the present invention is 
to provide an image processing apparatus, method and 
computer-readable memory which make it possible to 

10 extract a desired scene from a plurality of moving- 
picture materials in an efficient manner. 

According to the present invention, the foregoing 
object is attained by providing an image processing 
apparatus for processing a moving picture, comprising: 

15 frame extraction means for extracting frames 

constituting an entered moving picture; discrimination 
means for discriminating a scene change by comparing 
frames extracted by the frame extraction means; storage 
means for storing scene-change information relating to 

20 the scene change discriminated by the discrimination 

means; designating means for designating an image that 
corresponds to a scene that is the object of a search; 
comparison means for comparing a scene-change frame, 
which is obtained by referring to the scene-change 

25 information that has been stored in the storage means, 
and the image that has been designated by the 
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designation means; and scene extraction means for 
extracting a scene that corresponds to the image based 
upon result of the comparison performed by the 
comparison means . 
5 The foregoing object is attained by providing an 

image processing method for processing a moving picture, 
comprising: a frame extraction step of extracting 
frames constituting an entered moving picture; a 

C3 discrimination step of discriminating a scene change by 

i ^ 

iff 10 comparing frames extracted at the frame extraction step; 

O 

Ij a storage step of storing scene-change information 

Sj relating to the scene change discriminated at the 

Z~ discrimination step; a designating step of designating 

s :: 3 

an image that corresponds to a scene that is the object 
15 of a search; a comparison step of comparing a scene- 
change . frame , which is obtained by referring to the 
scene-change information that has been stored at the 
storage step, and the image that has been designated at 
the designation step; and a scene extraction step of 
20 extracting a scene that corresponds to the image based 
upon result of the comparison performed at the 
comparison step . 

The foregoing object is attained by providing a 
computer- readable memory storing program code of image 
25 processing for processing a moving picture, the memory 
including: program code of a frame extraction step of 
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extracting frames constituting an entered moving 
picture; program code of a discrimination step of 
discriminating a scene change by comparing frames 
extracted at the frame extraction step; program code of 
5 a storage step of storing scene-change information 
relating to the scene change discriminated at the 
discrimination step; program code of a designating step 
of designating an image that corresponds to a scene that 
is the object of a search; program code of a comparison 

If! 10 step of comparing a scene-change frame, which is 

1.3 

LJ obtained by referring to the scene-change information 

Hj that has been stored at the storage step, and the image 

that has been designated at the designation step; and 
program code of a scene extraction step of extracting a 
15 scene that corresponds to the image based upon result of 
the comparison performed at the comparison step. 

Other features and advantages of the present 
invention will be apparent from the following 
description taken in conjunction with the accompanying 
2 0 drawings, in which like reference characters designate 
the same or similar parts throughout the figures 
thereof . 
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BRIEF DESCRIPTION OF THE DRAWINGS 



Fig. 1 is a block diagram illustrating the 
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construction of an image processing apparatus according 
to an embodiment of the present invention; 

Fig. 2 is a block diagram illustrating the 
functional construction of an image processing apparatus 
5 according to an embodiment of the present invention; 

Fig. 3 is a flowchart illustrating the procedure of 
scene-change detection processing according to an 
embodiment of the present invention; 
□ Fig. 4 is diagram illustrating an example of block 

Iff 10 division of a frame according to an embodiment of the 

Q 

yj present invention; 

Fig. 5 is diagram illustrating an example of scene- 

; i 3 

change information according to an embodiment of the 
present invention ; 
15 Fig. 6 is a flowchart illustrating the procedure of 

scene extraction processing according to an embodiment 
of the present invention; and 

Fig. 7 is a diagram showing an example of a list of 
extracted scenes according to an embodiment of the 
20 present invention. 

DESCRIPTION OF THE PREFERRED EMBODIMENT 

A preferred embodiment of the present invention 
25 will now be described in derail with reference to the 
accompanying drawings . 



rii 



3 



- 5 - 



Fig. 1 is a block diagram illustrating the 
construction of an image processing apparatus according 
to a preferred embodiment of the present invention. 

As shown in Fig. 1, the apparatus includes a CPU 
5 101 for implementing various control operations 

regarding various components of the image processing 
apparatus; a ROM 102 for storing a booting program, 
which is run when the image processing apparatus is 
Q started up, and various data; a RAM 103 which stores a 

LO 10 control program run by the CU 101 and which provides a 

y working area when various control operations are 

Sj executed; a keyboard 104 and a mouse 105 for providing 

i.A ! 

: ; , an environment that allows a user to make various 

ffs inputs; an external storage device 106 constituted by a 

15 hard disk, floppy disk or CD-ROM; a display unit 107 

constituted by a CRT or LCD or the like which displays 
the results of processing for the user; a network 
interface controller (NIC) 108 for making possible 
communication with various devices on a network; a video 
20 interface (I/F) 109 that makes it possible to capture a 
moving picture from a video camera 110 or VTR 112; and a 
bus 111 for interconnecting the above-mentioned 
components . 

It should be noted that the video camera 110, VTR 
2 5 112 and external storage device 106 may be replaced by 
components connected to the network. 
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The functions of the image processing apparatus 
according to this embodiment will be described with 
reference to Fig. 2. 

A moving-picture input unit 201 is for inputting a 
5 moving picture captured from the video camera 110 or VTR 
112 via the video interface 109 or a moving picture 
downloaded from the network via the NIC 108. A frame 
extraction unit 202 successively extracts frames 
C.3 (images) , which constitute the entered moving picture, 

U1 10 in regular order. A frame storage unit 2 09 stores 

E t:c 7 

Id frames extracted by the frame extraction unit 202. A 

SI scene-change discrimination unit 203 examines each frame 

m 

to discriminate a scene change in a moving picture using 
an interf rame-similarity calculation unit 204. The 
15 latter calculates the degree of similarity between 

frames. A scene-change information storage unit 2 05 
stores information relating to a scene changes that has 
been discriminated by the scene-change discrimination 
unit 203. A pattern image designation unit 206 allows 
20 the user to designate a pattern image using the keyboard 
104 and mouse 105. A scene extraction unit 207 uses the 
interf rame-similarity calculation unit 204 to compare 
the pattern image that has been designated by the 
pattern image designation unit 2 06 with a frame, in 
25 which a scene change has occurred, obtained by referring 
to the scene-change information storage unit 205, and 
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extracts a desired scene corresponding to the pattern 
image . An extracted-scene list display unit 208 causes 
the display unit 107 to display a list of information 
relating to scenes extracted by the scene extraction 
5 unit 207. 

An example of operation of the image processing 
apparatus having the scene extraction function of this 
embodiment arranged as set forth above will now be 
described. 

10 [Scene-change detection processing] 

Processing for detecting a change in scene will be 
described with reference to Fig. 3. 

Fig. 3 is a flowchart illustrating the procedure of 
scene-change detection processing according to this 
15 embodiment of the present invention. 

First, a moving picture to undergo processing is 
input by the moving-picture input unit 201 at step S301. 
Next, at step S302, the frame extraction unit 202 
executes processing for extracting one frame at a time 
20 from the entered moving picture. Then, at step S303, 
the frame extraction unit 2 02 stores the extracted 
frames in the frame storage unit 209. 

It is determined at step S304 whether all frames of 
the entered moving picture have been extracted. If all 
25 frames have not been extracted ("NO" at step S304), 

steps S302 and S3 03 are repeated. If all frames have 
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been extracted ("YES" at step S304), then control 
proceeds to step S305. 

Step S305 calls for the scene-change discrimination 
unit 203 to use the interf rame- similarity calculatipn 
5 unit 204 to perform, in numerical order, an inter frame- 
similarity calculation between a frame of interest and a 
group of past frames. There are instances where the 
group of past frames is a single frame or a plurality of 

□ frames, and there is no particular limitation upon the 

Ul 10 algorithm. 

Ly The simplest example will now be described. First, 

SJ the immediately preceding frame and the current frame 

m 

w are each divided into a plurality of blocks vertically 

pjl and horizontally, as shown in Fig. 4. Next, average 

i 'zs 

! '« 15 values of R, G and B are calculated in regard to each 

! ::f block. This is followed by obtaining the sum of the 

Q 

squares of the differences between the corresponding R, 
G and B channels in the preceding and current frames. 
This sum of the squares is taken as interframe 

20 similarity, which represents the degree of intensity of 
the scene change. The smaller the value of the sum of 
the squares, the greater the similarity between frames; 
the larger the value, the smaller the similarity. In 
other words, this is a method of obtaining an instance 

2 5 in which the possibility of a scene change is high. A 
example of the calculation formula is as follows: 
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2 [ (Pl iR -P2 iR ) 2 + (Pli G -P2i G ) 2 + <PliB-P2i B > 2 ] 

i=l 

where i represents the block undergoing processing, and 
K : number of blocks into which the frame is 
divided; 

5 PliR: average value of R channel of ith block of 

preceding frame; 
PliG : average value of G channel of ith block of 
preceding frame; 

□ 

E .3 PliB : average value of B channel of ith block of 

..F§ 

□ 10 preceding frame; 

-J 

= P2±r: average value of R channel of ith block of 

"•\s 

•■*n current frame; 

P2ig: average value of G channel of ith block of 
j current frame; 

£ 15 P2iB : average value of B channel of ith block of 

«3 current frame 

On the basis of the result of calculation of 
interframe similarity, the scene-change discrimination 
unit 2 03 determines whether or not there has been a 
20 scene change. If there has not been a scene change 

("NO" at step S306) , control proceeds to step S308. If 
there has been a scene change ("YES" at step S306) , on 
the other hand, then, at step S307, the ID of the 
entered moving picture, the ID of the scene change and 
25 the frame are stored as the scene starting frame and the 
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similarity (or distance) representing the degree of 
intensity of the scene change is stored. These items of 
information shall be referred to as "scene-change 
information" . An example of scene-change information is 
5 illustrated in Fig. 5. This information is stored in 
the scene-change information storage unit 205. 

Next, at step S308, it is determined whether all 
frames of the moving picture have been discriminated for 
□ scene change. If all frames have not undergone scene- 

Iff 10 change discrimination ("NO" at step S308) , processing 

LJ from step S305 to step S307 is repeated until all scene- 

change frames of the moving picture and the similarity 
^' information are obtained. If all frames have undergone 

scene-change discrimination ("YES" at step S308) , on the 

! U= 
i - 

•~'2 15 other hand, then processing is exited. 

•-3 The foregoing is processing for detecting scene 

changes and is applied to all moving pictures that are 
to undergo scene extraction. 

[Scene extraction processing] 
20 Reference will be had to Fig. 6 to describe 

processing for designating a pattern image and 
extracting a desired scene. 

Fig. 6 is a flowchart illustrating the procedure of 
scene extraction processing according to this embodiment 
25 of the present invention. 

The user designates a pattern image using the 
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pattern image designation unit 206 at step S501. The 
pattern image designated is the image that represents 
the lead frame of a scene (e.g., the title screen of a 
specific part of a TV program) . One conceivable method 
of designating the pattern image is for the user to 
create an image that resembles the lead frame of the 
scene. Next, at step S502, the scene extraction unit 
207 regards the pattern image, which has been designated 
by the pattern image designation unit 206, as being a 
frame, and uses the interf rame-similarity calculation 
unit 204 to calculate the degree of similarity between 
this frame and the already detected scene-change frame. 
Though the method of calculating similarity is not 
particularly limited, one conceivable method is that 
mentioned in the scene-change detection processing 
described above. The scene extraction unit 207 extracts 
the scene-change frame from the frame storage unit 209 
upon referring to the ID of the moving picture and the 
starting frame of the scene-change information in the 
scene-change information storage unit 205. 

Next, at step S503, the scene extraction unit 2 07 
determines whether the pattern image is a frame 
identical with the scene-change frame based upon the 
degree of similarity calculated by the interframe- 
similarity calculation unit 204. If the frames are not 
identical ("NO" at step S503), control proceeds to step 
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5505. If the frames are identical ("YES" at step S503) , 
on the other hand, then, at step S504, the ID of the 
moving picture is output to the . extracted-scene list 
display unit 208 as well as the scene-change frame, 
which serves as the starting frame of a scene, and the 
frame (which will serve as the end frame of the scene) 
that immediately precedes the next scene-change frame. 
An example of a list of the extracted scenes is shown in 
Fig. 7. 

Next, at step S505, it is determined whether all 
scene-change frames have been compared with the pattern 
image. If all scene-change frames have not been 
compared with the pattern image ("NO" at step S505), 
steps S502 to S504 are repeated to extract scenes that 
begin with those frames of all scene-change frames that 
are identical with the pattern image. If all scene- 
change frames have been compared with the pattern image 
("YES" at step S505), then control proceeds to step 

5506. This is followed by step S506, at which the 
extracted-scene list display unit 208 causes the list of 
extracted scenes (Fig. 7) to be displayed for the user. 

This concludes scene extraction processing. 

Thus, in accordance with this embodiment, as 
described above, scene changes in an input moving 
picture can be detected automatically and the user, 
merely by designating a pattern image, can automatically 
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extract a frame of a scene, which corresponds to the 
pattern image. As a result, the user need not perform 
such operations as detection of scenes and creation of a 
time-code list, thus making it possible to perform 
5 operations such as editing in an efficient manner. 
[Other embodiments] 

In the embodiment described above, all frames are 
extracted and stored through steps S3 02 to S3 04, after 
C3 which discrimination of a change in scene is performed 

a 

I/I 10 at steps S305 to S308. However, it is permissible to 

r5 

Lj adopt an arrangement in which discrimination of scene 

Hj change is performed successively whenever a frame is 

; , : extracted, and only the necessary frames (the scene- 

O 

pl change frames, etc.) are stored in memory. 

: *~ 15 Further, though the list of extracted scenes (Fig. 

7) per se is displayed for the user, it may be so 
arranged that the extracted scenes are edited into a 
single moving picture based upon the scene list and the 
scene list is displayed for the user. Alternatively, a 
2 0 printer may be connected to the apparatus to output the 
moving picture on a recording medium. 

Further, a method in which the user creates the 
pattern image has been described as the method of 
designating the pattern image. However, it is 
25 permissible to adopt an arrangement in which scene- 
change frames extracted in scene-change detection are 
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displayed for the user and the user selects a frame from 
these selected frames as the pattern image. It may also 
be so arranged that the selection is made from a group 
of other images that have been prepared in advance. 
5 Further, though only one scene that starts from. a 

frame resembling the pattern image is extracted in the 
foregoing embodiment, it may be so arranged that the 
user can designate the number of scenes, thereby making 
□ it possible to extract a designated number of scenes 

Lfl 10 that start with a frame resembling the pattern image, 

hj It is permissible to designate a number of frames with 

regard to frames prior to and with regard to frames on 
!;!'" and after a frame resembling the pattern image. 

Further, though only one scene that starts from a 



ry 

15 frame resembling the pattern image is extracted in the 



foregoing embodiment, it may be so arranged that the 
user can designate scene length (time) , thereby making 
it possible to deduce time from the number of frames of 
a scene based upon scene-change information and extract 

20 a scene of a designated length that begins with the 

frame resembling the pattern image. It is permissible 
to designate scene length (time) with regard to frames 
prior to and with regard to frames on and after a frame 
resembling the pattern image. 

25 Only one scene that starts from a frame resembling 

the pattern image is extracted in the foregoing 
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embodiment. However, if the frame immediately preceding 
a scene-change frame is extracted from the scene-change 
information and compared with the pattern image at step 
S502, then it will be possible to extract a scene that 
5 ends with a frame resembling the pattern image. 

Again, only one scene that starts from a frame 
resembling the pattern image is extracted in the 
foregoing embodiment. However, if the pattern image is 
□ compared with all frames, a similar frame is found and a 

Iff 10 scene in which this frame is present is obtained by 

□ 

hf referring to the scene-change information, then it will 



be possible to extract a scene containing a frame 
resembling the pattern image. 

As described above, this embodiment enables user to 
15 automatically extract a frame, which corresponds to the 
pattern image, that is the leading, intermediate or 
final frame of a scene. In addition, extracted scenes 
can be combined into a single moving picture 
automatically. 

20 The present invention can be applied to a system 

constituted by a plurality of devices (e.g., a host 
computer, interface, reader, printer, etc.) or to an 
apparatus comprising a single device (e.g., a copier or 
facsimile machine, etc.). 

25 Furthermore, it goes without saying that the object 

of the invention is attained by supplying a storage 
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medium storing the program codes of the software for 
performing the functions of the foregoing embodiments to 
a system or an apparatus, reading the program codes with 
a computer (e.g., a CPU or MPU) of the system or 
apparatus from the storage medium, and then executing 
the program codes . 

In this case, the program codes read from the 
storage medium implement the novel functions of the 
invention, and the storage medium storing the program 
codes constitutes the invention. 

Further, the storage medium, such as a floppy disk, 
hard disk, optical disk, magneto-optical disk, CD-ROM, 
CD-R, magnetic tape, non-volatile type memory card or 
ROM can be used to provide the program codes. 

Furthermore, besides the case where the aforesaid 
functions according to the embodiments are implemented 
by executing the program codes read by a computer, it 
goes without saying that the present invention covers a 
case where an operating system or the like running on 
the computer performs a part of or the entire process in 
accordance with the designation of program codes and 
implements the functions according to the embodiments. 

It goes without saying that the present invention 
further covers a case where, after the program codes 
read from the storage medium are written in a function 
expansion board inserted into the computer or in a 



- 17 - 




memory provided in a function expansion unit connected 
to the computer , a CPU or the like contained in the 
function expansion board or function extension unit 
performs a part of or the entire process in accordance 
5 with the designation of program codes and implements the 
function of the above embodiment. 

In a case where the present invention is applied to 
the above-mentioned storage medium, program code 
4? corresponding to the flowcharts of Figs. 3 and 6 

y 10 described earlier would be stored on the storage medium. 

UJ Thus, in accordance with the embodiments of the 

HI present invention as described above, it is possible to 

:..f r 

provide an image processing apparatus, method and 
f : y computer-readable memory through which a desired scene 

'jj 15 can be extracted from a plurality of moving-picture 

materials in an efficient manner. 

As many apparently widely different embodiments of 

the present invention can be made without departing from 

the spirit and scope thereof, it is to be understood 
20 that the invention is not limited to the specific 

embodiments thereof except as defined in the appended 

claims . 
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